r 

ACT Research and Policy 

Research Reports 
Educator Reports 
Policy Reports 
Issue/Information Briefs 

Student- and Teacher-Reported 
Behavioral Measures: Do They Agree? 

r 

Issue Brief 

April 2013 


• Middle school academic 
behaviors account for about 
30% of the prediction of early 
high school GPA (ACT, 2011). 

• Measuring behaviors can help 
schools identify and intervene 
with students at risk of academic 
failure or dropout. 

• ACT Engage® Grades 6-9 
assessment measures student 
behaviors from the student’s 
perspective (ACT, 2012b). 

• ACT Engage® Teacher Edition 
measures student behaviors from 
the teacher's perspective (ACT, 
2012c). 


Middle school students’ behavior is an important dimension of high 
school readiness (ACT, 2008). Research has shown that academic 
achievement and academic behavior in middle school affect 
high school academic outcomes (ACT, 2011; Casillas et al. , 2013; 
Langenkamp, 2010; Tobin and Sugai, 1999), high school persistence 
(ACT, 2012a), and high school graduation (McLeod and Kaiser, 2004; 
Karakus et al., 2012). McIntosh (2008) showed that the behavior of 
grade 8 and 9 students has a significant effect on their academic skills 
and achievements. The importance of academic behaviors goes beyond 
high school — other studies show that academic behaviors predict 
success and persistence in college (Robbins et al., 2006; Conrad, 2006; 
Wang, 2013; US Department of Education, 2001). Research has also 
shown that behavior during adolescence is related to employment, 
wages in adulthood (Bowles et al., 2001), and labor market outcomes 
(Heckman et al., 2006). 

The academic behaviors that are important for success in middle 
school, high school, and beyond can be grouped into three broad 
domains (Robbins et al., 2004): 

Motivation includes personal characteristics that help students to 
succeed academically by focusing and maintaining energies on goal- 
directed activities; 

Social Engagement includes interpersonal factors that influence 
students’ successful integration into their environment; and 

Self-Regulation includes the thinking processes and emotional 
responses that govern how well students monitor, regulate, and 
control their behavior related to school and learning. 

Middle school student behavior can be assessed from multiple 
perspectives, including that of the students themselves (ACT, 2012b; 
Montague et al., 2011) and that of teachers (ACT, 2012c; Montague et 
al., 2011). This raises questions about the accuracy of each of these 
perspectives and how well they predict later outcomes. 
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• For each domain and each 
perspective, students were 
grouped according to their 
score ranks: 

Low (bottom 25%) 
Medium (middle 50%) 
High (top 25%) 


This paper addresses three key questions about student and teacher 
observations of behavior. 

1. Do both types of behavioral measures — student-reported and 
teacher-reported — predict important educational outcomes: course 
grades, absenteeism, and being suspended from school? Does using 
both perspectives together improve prediction? 

2. How similar are the students’ self-reports to teachers’ perceptions of 
these same students’ behaviors in each of the three domains? 

3. Do the relationships between student-reported and teacher-reported 
behavioral measures vary by student grade level? 

Data were collected for over 6,000 students from 42 schools. The 
student-reported behavioral measures were collected using ACT 
Engage Grades 6-9, an instrument designed to measure behavior and 
psychosocial factors. Behavioral ratings were collected from these 
students’ teachers using ACT Engage Teacher Edition. 1 

To what extent are student and teacher behavioral 
measures related to educational outcomes? 

The charts on the following pages show the relationships between 
student- and teacher-reported behavioral measures and three 
educational outcomes. 

Academic Success was defined as earning grades of mostly Bs or As. 

Absenteeism was measured by the number of days absent from school 
in the past month. 

Suspension was defined as having ever been suspended from school. 

The three outcomes are based on student-reported data collected at the 
same time students took the ACT Engage Grades 6-9 assessment. For 
analysis, each behavioral domain was paired with a theoretically-linked 
outcome: motivation with academic success, social engagement with 
absenteeism, and self-regulation with suspension. 

For each student, we calculated teacher-reported and student-reported 
summary measures for each domain. Then, students were classified 
according to their rank on each measure: Low (bottom 25%), Medium 
(middle 50%), and High (top 25%). 


1 The measurements from both instruments can be mapped to the three behavioral domains 
introduced above (motivation, social engagement, and self-regulation). Each instrument’s scales 
and their mappings to the three domains are presented in the Appendix. 
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As shown in the following three graphs, for all three behavioral 
domains, both the student- and teacher-reported measures of behavior 
were significant predictors of the relevant educational outcomes. 

This provides evidence that while students’ perceptions of their own 
behavior may differ somewhat from those of their teachers, both 
perspectives are important for understanding risk. In addition, for all 
three domains, the combination of both teacher- and student-reported 
behavior provided significantly better prediction of the outcomes than 
either perspective alone. 


Motivation and Academic Success 

Low Teacher Rating "Medium Teacher Rating "High Teacher Rating 



Graph reads: Among students with low student and low teacher assessments of motivation, 46% earned 
grades of mostly Bs or mostly As. 


• Academic success rates ranged 
from 46% for students with low 
scores from both perspectives to 
99% for students with high scores 
from both perspectives. 

• The teacher-reported measure 
of motivation was more strongly 
related to academic success 
than the student self-report. 
However, using both measures 
of motivation predicts academic 
success better than using either 
the student-reported score or the 
teacher rating alone. 2 

• Results show that both student 
and teacher perspectives of 
student motivation are important 
for understanding academic 
outcomes. 


2 The Wald Chi-Square statistics (and R-Square) from a logistic regression model predicting 
academic success were 686.3 (and 0.136) for the teacher-reported rating alone, 461.8 (and 0.085) 
for the student-reported score alone, and 811.9 (and 0.167) for the combined teacher rating and 
student score. Tests of the difference between using the combined predictors — student-reported 
score and teacher rating — and using each of the two individual predictors alone were statistically 
significant with p-value < 0.05. 
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• Average days absent in the 
past month ranged from 2.6 for 
students with low scores from 
both perspectives to 1 .1 for 
students with high scores from 
both perspectives. The difference 
of 1 .5 days per month translates 
to about 1 3-1 4 additional days 
per school year missed. 

• Absenteeism increased with lower 
levels of student- and teacher- 
reported measures of social 
engagement. 

• The teacher- and student- 
reported measures of social 
engagement were similar in the 
strength of their relationships 
with absenteeism. Using both 
the student-reported score 
and the teacher rating of social 
engagement provides a better 
predictor of absenteeism than 
either measure alone. 3 

• For students with high self-reports 
of social engagement, there was 
little variation in absenteeism 
across the teacher-reported levels 
(see three bars on the far right of 
the chart). 


Social Engagement and Absenteeism 


Low Teacher Rating L Medium Teacher Rating "High Teacher Rating 



Low Student Score Medium Student Score High Student Score 


Graph reads: Among students with a medium student- and low teacher-assessments of social engagement, 
the average number of school days missed in the past month was 2.2. 


3 The F statistics (and R-Square) from a regression model predicting number of days absent were 
87.2 (and 0.027) for the teacher-reported rating alone, 77.0 (and 0.024) for the student-reported 
score alone, and 67.4 (and 0.041) for the combined student score and teacher rating. Tests of the 
difference between the combined predictors — student-reported score and teacher rating — and 
each individual predictor alone were statistically significant with p-value < 0.05. 
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Self-Regulation and Suspension 

Low Teacher Rating "Medium Teacher Rating "High Teacher Rating 

70% -i 


• Student- and teacher-reported 
measures of self-regulation 
were both related to school 
suspension. 



• The student-reported measure of 
self-regulation was more strongly 
related to suspension. However, 
using both the student score and 
the teacher rating of a student’s 
self-regulation better predicts 
whether or not the student has 
been suspended. 4 

• Suspension rates ranged from 
53% for students with low scores 
from both perspectives to 0% for 
students with high scores. 


Graph reads: Among students with a high student and medium teacher assessments of self-regulation, 5% 
had been suspended from school. 


• Very few students with high 
teacher ratings of self-regulation 
had ever been suspended from 
school. 


4 The Wald Chi-Square (and R-Square) statistics from a logistic regression model predicting 
suspension were 427.4 (and 0.077) for the student-reported level alone, 332.8 (and 0.158) for the 
teacher-reported rating alone, and 508.2 (and 0.181) for the combined student score and teacher 
reported rating. Tests of the difference between the combined predictors — student-reported score 
and teacher rating — and each individual predictor were statistically significant with p-value < 0.05. 
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• Student-teacher score 
correlations: 

Motivation: 0.41 
Social Engagement: 0.30 
Self-Regulation: 0.41 

• Overall agreement 5 : 

Motivation: 49% 

Social Engagement: 44% 
Self-Regulation: 50% 


Do student and teacher behavioral 
measures agree? 

We found that student-reported scores are predictive of teacher ratings 
for all three domains, but the agreement is not perfect. The table below 
shows the percentage of students at each teacher-reported rating 
level (Low, Medium, and High) for each of the three student-reported 
score levels. For example, 41% of students with a high self-reported 
motivation score also received a high rating from their teachers for 
this domain. Similarly, 30% of students with a low self-reported self- 
regulation score also received a low rating from their teacher. 


Student and Teacher Assessments by Domain 


Motivation 

Student Score 


Teacher Rating Level (%) 


Level 

Low 

Medium 

High 

Total 

Low 

38.6 

52.5 

8.9 

100.0 

Medium 

20.0 

58.5 

21.6 

100.0 

High 

7.6 

51.4 

41.0 

100.0 

Social Engagement 

Student Score 


Teacher Rating Level (%) 


Level 

Low 

Medium 

High 

Total 

Low 

28.8 

54.8 

16.4 

100.0 

Medium 

15.9 

53.7 

30.4 

100.0 

High 

9.8 

45.9 

44.2 

100.0 

Self-Regulation 

Student Score 


Teacher Rating Level (%) 


Level 

Low 

Medium 

High 

Total 

Low 

30.0 

60.2 

9.8 

100.0 

Medium 

12.2 

64.4 

23.4 

100.0 

High 

3.7 

50.9 

45.4 

100.0 


Based on data from 6,326 students from 42 schools. The students took ACT Engage Grades 6-9 and 
teachers rated their behavior using ACT Engage Teacher Edition. Data were collected for students in 
grades 6-9. 


5 Defined as the percentage of students whose Low/Medium/High classifications matched. 
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How does the relationship between student- 
reported and teacher-reported behavioral 
measures vary by student grade level? 

The study included students in grades 6 through 9, allowing us to 
examine whether the relationship between student- and teacher- 
reported behavioral measures varied by grade level. The following 
chart shows the relationship between average teacher ratings on 
the motivation scale and student-reported motivation scores. For 
each student grade level, a trend line was created to summarize the 
relationship between these two scores. 6 The next two graphs show 
similar results for social engagement and self-regulation. 

Mean Teacher Ratings by Student Score: Motivation 



^Grade 6 
““Grade 7 
““Grade 8 
— Grade 9 


• On average, the relationship of 
student-reported and teacher- 
reported measures of students’ 
motivation was nearly identical 
for students in grades 6, 7, and 
8. The discrepancy between 
student and teacher perceptions 
of students’ motivation is larger in 
9th grade. 


The teacher ratings of motivation (and social engagement and self- 
regulation) increase steadily as the student-reported motivation score 
increases. At the highest levels of student-reported motivation, the 
average teacher rating tops out between 6 and 7 (on the 8-point rating 
scale). 


6 The relationship between student-reported and teacher-reported scores for each grade level is 
summarized with linear trend line in all three graphs. 
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• On average, teacher ratings of 
students’ social engagement 
increase with student-reported 
scores. Relative to students in 
grades 6-8, teachers gave 9th 
grade students lower social 
engagement ratings. 


Mean Teacher Ratings by Student Score: Social Engagement 


8 i 
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■“Grade 6 
•“Grade 7 
“■Grade 8 
— Grade 9 


20 30 40 50 60 

Student Social Engagement Score 


• The relationship of student- 
reported and teacher-reported 
measures of students’ self- 
regulation was nearly identical for 
students in grades 6, 7, and 8. 
The discrepancy between student 
and teacher perceptions of 
students’ self-regulation is larger 
in 9th grade. 


• Across all grade levels and 
domains, teacher ratings 
increased steadily as student- 
reported behavioral scores 
increased. 

• Generally, teacher ratings were 
lower for 9th grade students 
compared to 6th, 7th, or 8th 
grade students. 


Mean Teacher Ratings by Student Score: Self-Regulation 




•“Grade 6 
■“Grade 7 
— Grade 8 
— Grade 9 
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Student Self-Regulation Score 


The relationships between student-reported and teacher-reported 
behaviors were similar across all grades with the exception of grade 9. 
For social engagement and self-regulation, students in grade 9 tended 
to receive lower teacher ratings 7 than students in grades 6 through 8. 
On average, 9th grade students with higher motivation scores were 
rated by their teachers as less motivated than students in grades 6 
through 8. 8 


7 A Tukey-adjusted multiple comparisons test of pairwise differences between means of teacher 
ratings showed significantly (p-value < 0.05) smaller grade 9 means compared to the means of 
grade 6, 7, and 8. 

8 For students’ motivation scores above 35, a Tukey-adjusted multiple comparisons test of pairwise 
differences between means of teacher ratings showed a significantly (p-value < 0.05) smaller 
grade 9 mean compared to the means of grade 6, 7, and 8. There was no significant difference in 
grade level means of teacher ratings for student motivation scores below 35. 
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Implications for School Practice and Policy 

The three broad academic behavior domains of focus in this study — 
motivation, social engagement, and self-regulation — are important 
contributors to educational and workplace success. The results of 
this study suggest that the assessment of academic behaviors from 
multiple perspectives — student and teacher — yields more accurate and 
actionable information than either perspective alone. Implications for 
classroom and school practice and policy focus on more effective use 
of student information to improve student educational outcomes. 

1. In addition to traditional academic measures such as test scores 
and grades, it is important to examine behavioral factors for better 
signaling of student risk. Both student-reported scores and teacher- 
reported ratings are related to important educational outcomes, such 
as earning good grades, absenteeism, and school suspension. 

2. Teacher observations of student behavior, and their relationships 
to educational outcomes, are an important source of input that is 
complimented by student self-assessments of their own behavior. 
While student and teacher assessments of student behavior do not 
completely agree, our analyses show that each is incrementally 
predictive of educational outcomes. In addition, comparing student- 
reported and teacher-reported behavioral measures can: help 
clarify behavioral expectations; confirm behavioral signals; provide 
additional information to educators about student strengths and 
weaknesses; and lead to more targeted instruction for students. 

The study results also illuminate a 9th grade perception gap related to 
student behaviors, reinforcing the importance of efforts to improve the 
transition from middle school to high school. The relationship between 
student scores and teacher ratings varies by student grade level for 
self-regulation and social engagement. For a given self-reported score, 
teachers tend to rate grade 9 students lower than they rate students 
in grades 6-8. For motivation, high scoring grade 9 students are rated 
less motivated than their counterparts in grades 6 through 8. Typically, 
grade 9 in the United States is located within the first year of the high 
school context, when students are expected to assume substantially 
more responsibility and control over their educational experience 
and success. 
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Appendix 


ACT Engage Scales by Domain 


Domain Name 

ACT Engage 
Grades 6-9 Scales 

ACT Engage 
Teacher Edition Scales 

Motivation 

Academic Discipline 
Commitment to School 
Optimism 

Initiative 

Planning & Organizing 
Sustained Effort 
Performance 

Social Engagement 

Family Attitude toward 

Education 

Family Involvement 

Relationships with School 
Personnel 

School Safety Climate 

Communication 
Working with Others 

Self-Regulation 

Managing Feelings 
Orderly Conduct 
Thinking Before Acting 

Managing Feelings 
Orderly Conduct 
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